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PATENT 

Attorney Docket No. 17509-713 


IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of 

Hill Branscomb 

Application No.: Not yet assigned 

(Rule 60 Continuation of 08/243,046) 
Filed: Herewith (September 5, 1997) 

For: APPARATUS AND METHOD FOR 
ASSEMBLING CONTENT 
ADDRESSABLE VIDEO 

UTILITY PATENT 
TRANSMITTAL OF FILING UNDER 37 CFR 1.60 

Box Patent Application 

Assistant Commissioner for Patents 

Washington, D.C. 20231 

Dear Sir: 

This is a request for filing a [X] Continuation [ ] Divisional application under 37 CFR 
1 .60, of pending prior Application No. 08/243.046 filed on Mav 16, 1994 by Hill Branscomb for 
APPARATUS AND METHOD FOR ASSEMBLING CONTENT ADDRESSABLE 
VIDEO . 

1. Application 
Enclosed are: 

[X] a true copy of the complete prior application as filed including drawings; 

34 pages of specification, claims and abstract, and 

6 sheet(s) of [ ] formal [X] informal drawing(s); 
[X] a copy of the Declaration as filed in the prior application; 
[X] copies of the Assignment(s) as filed in the prior application; 


Prior Application 
Group Art Unit: 2415 
Examiner: M. Luu 
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[X] 1 verified statement claiming small entity status as filed with the prior application; 
[X] other: Supplemental Declaration of inventor as filed in parent application; 

Preliminary Amendment filed January 6, 1993 and cited 
in Supplemental Declaration by inventor. 

Declaration under 37 CFR 1.132 of Michael T. MacKay 

2. Amendment 

[X] Amend the specification by inserting before the first line the sentence: -This 
application is a [X] continuation, [ ] divisional, of copending Application No. 
08/243 ,046, filed May 16, 1994, now allowed, which is a continuation of 
Application No. 08/146,400, filed November 1, 1993, now abandoned, which is a 
continuation in part of Application No. 08/000,927, filed January 6, 1993, now 
abandoned, which is a continuation in part of Application No. 07/640,489, filed 
December 8, 1989, now abandoned. - 

[ ] Cancel in this application original claims of the prior application before 

calculating the filing fee. 

[X] A preliminary amendment is enclosed. The enclosed Preliminary Amendment 
was originally filed January 6, 1993 in parent application Serial No. 08/000,927, 
and is referred to in the Supplemental Declaration by the inventor attached hereto. 
Entry of this Preliminary Amendment is requested. 

[ ] Delete the following inventor(s) name(s) from this application: 


3. Power of Attorney 

[X] The power of attorney in the prior application is to: Mark A. Haynes 


Paul Davis 
Mark A. Haynes 
Charles C. Cary 
Michael J. Murphy 
Michael J. Panepucci 
David J. Weitz 
Kent R. Richardson 
George A. Willman 
Nancy Gamburd 
John J. Bruckner 
U.P. Peter Eng 


29,294 

30,846 

36,764 

37,404 

37,203 

38,362 

39,443 

P-41,378 

38,147 

35,816 

39,666 


[X] 
[] 


[] 
[] 


The power appears in the original papers in the prior application. 

Since the power does not appear in the original papers, a copy of the power in the 

prior application is enclosed. 

A new power has been executed and is attached. 

Recognize as Associate Attorney(s): 
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4. Fee Calculation The fee has been calculated as follows: 


CLAIMS 



NO. OF 
CLAIMS 


EXTRA 
CLAIMS 

RATE 

FEE 

Basic Applicati 

on Fee 

/ /u.uu 

Total Claims 

13 

MINUS 20 = 

0 

$22.00= 

0.00 

Independent 
Claims 

4 

MINUS 3 = 

1 

$80.00= 

£0 no 

If multiple dependent claims are presented, add $260.00 

-0- 

Total Application Fee 

$850.00 

If verified statement claiming small entity status is enclosed, subtract 50% 
of Total Application Fee 

-425.00 

Add Recording Fee of $40.00 if Assignment document is enclosed 

40.00 

TOTAL APPLICATION FEE DUE 

$465.00 


[ ] A check in the amount of $ is enclosed. 

[X] Charge <S 465.00 to Deposit Account No. 23-2415 (Docket No. 17509-713). The 
Assistant Commissioner is authorized to charge any additional fees which may be 
required, or credit any overpayment to Deposit Account 23-2415. A duplicate 
copy of this paper is enclosed. 

[ ] Fees will be paid when responding to the Notice to File Missing Parts. 

Address all future communications to: 
Mark A. Haynes 

Wilson, Sonsini, Goodrich & Rosati 

650 Page Mill Road 

Palo Alto, CA 94304-1050 

Respectfully submitted, 

WILSON SONSINI GOODRICH & ROSATI 

Date: September 5. 1997 

650 Page Mill Road 
Palo Alto, CA 94304-1050 
(415) 493-9300 


By: 

Mark A. Haynes 
Registration No. 30,846 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 


In re Patent Application of 
HILL BRANS COMB 

Application No.: 08/000,927 

Filed: January 6, 1993 

For: APPARATUS AND METHOD FOR 
ASSEMBLING CONTENT 
ADDRESSABLE VIDEO 


PATENT 


Group Art Unit: 2 609 
Examiner : M . Luu 


DECLARATION OF MICHAE L T. MacKAY 
UNDER 37 C-F.R. S 1.132 


I, Michael T. MacKay, hereby declare as follows: 

1. An expert with over 15 years of experience in the design, 
implementation and operation of multi-media environments, including 
computer graphics and video. A copy of a recent resume for me is 
attached hereto as Exhibit 1. 

2. Over the course of my career, I have been involved in the 
hiring and management of teams of engineers, and have developed a 
knowledge of the ordinary level of skill in the video and computer 
graphics arts. Although, the ordinary level of skill may be 
somewhat higher, the minimum skills required are a degree in 
electrical engineering or computer science with at least one year of 
experience in film, video, and graphics engineering. 

3. In connection with the above-identified U.S. patent 
application, I have been informed that the Patent Examiner has 
rejected certain claims in the application, on the basis that the 
application fails to provide an adequate written description of the 
invention. I have reviewed the application and claims. It is my 
opinion that all elements of the claims as described in the 
application, and that the description is sufficient to enable a 
person of ordinary skill in the computer graphics art to practice 
the invention . 

4. Particularly, the Examiner has stated: 
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The Applicant has failed to disclose the exact 
"means for associating tags with frames of video 
data..." as recited in claims 1 and 7. How is the 
"associating means" coupled to the storage means 
since the drawings do not show the "associating 
tag means" as specified in the claims. (Page 3 of 
the Official Action mailed April 30, 1993) . 

5. The elements in claims 1 and 7 to which this rejection is 
directed, read as follows: 

In Claim 1: 

"means, coupled with the storage means, f or _ 
associating tags with frames of video data in the 
plurality, the tags indicating the contents of the 
video images defined by the associated frames;" 

In Claim 7 : 

"associating tags with frames of video data in the 
plurality, the tags indicating the contents of the 
video images defined by the associated frames;" 

6. In the specification, the "means for associating tags", 
and the step of associating tags with frames of video data is 
clearly described. In particular, in Fig. 4, step 401 reads " ASSIGN 
KEY TO EACH VIDEO FRAME" . A person of ordinary skill in the art 
would readily understand that this results in association of a tag 
(key) with each video frame. Although the term "key" is used in the 
figure, those of skill in the art will understand that a key 
assigned with a frame would be considered a "tag". 

7 . The process for generating the keys is described on page 
11, lines 9-15. 

8. Further, on page 15, lines 13-15 of the specification, 
the step of "stamp keys on frames (or other recording methods) ..." 
describes the associating of tags with frames of video data. 

9 . In terms of the manner of implementing the tags with the 
video frames, the specification clearly describes how to make and 
use the system of tagging frames. Beginning at page 18, line 20, 
continuing to page 19, line 8, the application describes the use of 
an expanded SEMPTE time code structure to provide information which 
indicates the content of the frame. 

10. Accordingly, the application describes a system in which 
tags are associated with frames of video, wherein the tags indicate 
the content of the frame to which they are associated. Further, the 
specification enables a person of ordinary skill in the art to 
create and use such tags. 

11. Furthermore, the specification clearly shows how the 
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means for associating tags is coupled to the storage means. In 
particular, the machine generating the video frames stamps the 
frames of video with a tag. The associating means is a routine 
executed by the computer. Fig. 1 of the specification shows the 
computer 100 connected to the video storage 103 and to the camera 
position control 101. Thus, it is absolutely clear that the 
computer 100 which associates the tags with the frames of video data 
is coupled with the video storage system, and persons of ordinary 
skill in would unquestionably understand that connection based on a 
review in the specification. 

12. The Examiner has also stated: 

"What exactly is the "associating position means", 
and how is this "associating position means" 
coupled to the "processing means" since the 
drawings do not show the "associating position 
means" as specified in the claims." (Page 3 of 
the Official Action mailed April 30, 1993) 

13. This rejection is addressed at the following elements of . 
claims 1 and 7 : 

In Claim 1 : 

"means, coupled to the processing means, for 
associating positions in the content video image with 
addresses of storage locations storing corresponding 
frames of video data." 

In Claim 7: 

"associating with data processing means the 
positions in the content video image with 
addresses of storage locations storing 
corresponding frames of video data." 

14. The specification describes these elements with reference 
to Fig. 4, elements 402 and 403. This figure describes a computer 
algorithm by which a content image is generated using the keys that 
have been assigned to each video frame, and by which the video 
frames are then compiled for addressing in response to the keys. 
These two steps are standard computer processes which persons of 
ordinary skill in the art will clearly understand as a means for 
associating the position of a video frame in a content image with 
the content of the video frame based on the key. Thus, the 
application describes the means for associating positions, and 
enables a persons of ordinary skill to make and use such means. 

15. The application also provides a clear example of an 
algorithm for creating the means for associating positions at page 
15, lines 22 through page 16, line 8. Particularly, the algorithm 
is based on the use of a data base having a position in the content 
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image of a key, and associating a key to a video frame by addresses. 
Thus, the position in the content image is associated with a key, 
and the key is associated with a frame of data. 

16. In am informed that the Examiner is taking the position 
that the "associating position means" is not shown in the drawings. 
Fig. 4 illustrates a computer program for accomplishing such tasks, 
and is therefore believed to show the claimed feature. Furthermore, 
it's clear that the computer program is executed by the computer 
which includes a processing means. Thus, the processing means in 
the computer is associated with or coupled with the means for 
associating positions, as all computer programs are coupled with 
their host processor. 

17. I have been informed that the Examiner has also stated: 

"What exactly is the "means for accessing the 
frames of video data", and how is this "means for 
accessing the frames of video data" coupled with 
the "position selecting means" and the means for 
associating positions" since the drawings do not 
show the "means for accessing the frames of video 
data". (Page 3 of the Official Action mailed 
April 30, 1993) . 

18. The means for accessing the frames of video data appear 
in claims 2 and 6 in a means plus function format, and in claim 8 in 
a process format. Review of the specification reveals that the 
means for selecting a position in the content video image is the 
input device 111 shown in Fig. 1. Page 7, line 14-line 18, 
describes the use of the user input device 111, "such as a mouse or 
track ball" to position a cursor icon 112 on the content image 106. 
Thus, the user selects a position on the content image using an 
input device as clearly described in the specification in the 
drawings . 

19. The "means for accessing the frames of video data" in 
Fig. 1 comprises a computer system 100. This system is expanded in 
Fig. 2, in which the computer 201 is used to control video disk 
players 202 and 203. The video disk players access stored frames of 
video in response to the computer control as well known in the art. 

20. Thus, it is submitted that the application describes the 
means for accessing frames of video, and the means for selecting a 
position on the content image. Further, the application enables 
those of ordinary skill in the art to practice the invention 
claimed. 

21. In summary, the invention described and claimed in the 
above identified U.S. patent application uses programming techniques 
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well known in the art to accomplish the unique features described. 
It is based on using an associative link between a content image, 
and a key or tag indicating the content of a video frame associated 
with the tag. The processing control can be easily implemented 
using data base or table look-up techniques well known in the 
computer graphics art. 

22. Overall, it is my opinion that all the elements claimed in 
the present application are described in the specification, and the 
description of such elements enables a person of ordinary skill in 
the art to practice the invention. 

I hereby declare that all statements made herein of my 
knowledge are true and that all statements made herein on 
information and belief are believed to be true, and further that 
these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize 
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IN THE UNITED STATES PATENT AND T RADEMARK OFFICE 
in re Application ) PATENT APPLICATION 

Inventor (s) : Hill Branscomb ) 

) 

Serial No. : 07/640,489 ) 

) 

Filed: 01/11/91 ) 

) 

Title: APPARATUS AND METHOD FOR ) 

ASSEMBLING CONTENT ADDRESSABLE VIDEO ) 

; 

VERIFIED STATEMENT CLAIMING SMALL ENTITY STATUS 
?7 C.F.R Sl.Off) AND 1.27(c) - SMALL BUSINESS CONCERN 

I hereby declare that I am: 

The owner of the small business concern identified below. 


X 


An official of the small business concern empowered to act on behalf of the 
concern identified below. 

Name: Advanced Interaction. Inc. _ 


Address: 2810 Fillmore Street. San Francisco . California 94123 

I hereby declare that the above identified small business concern qualifies 
as a small business concern as defined in 13 C.F.R. §121.3-18, and reproduced in 
37 C.F.R. § 1.9(d), for purposes of paying reduced fees under Section 41(a) and 
(b) of Title 35 U.S.C. in that the number of employees of the concern, including 
those of its affiliates, does not exceed 500 persons. For purposes of this 
statement, (1) the number of employees of the business concern is the average 
over the previous fiscal year of the concern of the persons employed on a full- 
time, part-time or temporary basis during each of the pay periods of the fiscal 
year, and (2) concerns are affiliates of each other when either, directly or 
indirectly, one concern controls or has the power to control the other, or a 
third-party or parties controls or has the power to control both. 

I hereby declare that rights under contract or law have been conveyed to 
and remain with the small business concern identified below with regard to the 
invention entitled: 

APPARATUS AND METHOD FOR ASSEMBLING 
CONTENT ADDRESSABLE VIDEO 

by inventor (s) : Hill Branscomb 

described in: 

X Application Serial No. 0 7/640.489 filed Ql/11/91 

If the rights held by the above-identified small business concern are not 
exclusive, each individual, concern or organization having rights to the 
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invention is listed below and no rights to the invention are held by any 
person, other than the inventor, who could not qualify as a small business 
concern under 37 C.F.R. §1.9(d) or by any concern which would not qualify as a 
small business concern under 37 C.F.R. §1.9(d) or a nonprofit organization under 
37 C. F.R. §1. 9(e). 


NAME:. 


ADDRESS: 


[ ] Individual [ ] Small Business Concern [ ] Nonprofit Organization 
NAME : _ — 


ADDRESS:. 


[ ] Individual [ ] Small Business Concern [ ] Nonprofit Organization 

I acknowledge the duty to file, in this application or patent, notification 
of any change in status resulting in loss of entitlement to small entity status 
prior to paying, or at the time of paying, the earliest of the issue fee or any 
maintenance fee due after the date on which status as a small business entity is 
no longer appropriate. (37 C.F.R. §1. 28(b)). 

I hereby declare that all statements made herein of my own knowledge are 
true and that all statements made on information and belief are believed to be 
true; and further that these statements were made with the knowledge that willful 
false statements and the like so made are punishable by fine or imprisonment, or 
both, under Section 1001 of Title 18 of the United States Code, and that such 
willful false statements may Jeopardize the validity of the application, any 
patent issuing thereon, or any patent to which this verified statement is 
directed . 

Name of Person Signing Hill Branscomb 

Title of Person Signing President _. 

Address of Person Si/nin p 2810 Filljtbre Street. San Francisco. CA 9412 


Signature^ 


Date March 5. 1991 


Note: Separate verified statements are required from each named person, 
concern or organization having rights to the invention averring to their status 
as small entities. (37 C.F.R. §1.27). 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 


In re Application 


PATENT APPLICATION 


Inventor: 


Hill Branscomb 


Art Unit: 2609 


SC/Serial No.: 


07/640,489 


Examiner: M. Luu 


Filed: January 11, 1991 


Title: APPARATUS AND METHOD FOR 

ASSEMBLING CONTENT ADDRESSABLE 
VIDEO 


CERTIFICATE OF MAILING BY 'EXPRESS MAIL" UNDER 3 7 C.F.R. S 1.10 

"Express Mail" mailing label number: RB712972183US 
Date of Mailing: January 6, 1993 

I hereby certify that this correspondence is being deposited with the United 
States Postal Service, utilizing the "Express Mail Post Office to Addressee" service 
addressed to Commissioner of Patents and Trademarks, Washington, D.C. 20231 and 
mailed on the above Date of Mailing with the above "Express Mail" mailing label 
number. 


Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

Dear Sir: 

Entry of the following Preliminary Amendment is requested in the File Wrapper 
Continuation filed herewith. 


In the Specification 

On page 25, line 11, after "step one", please insert the following: 

~ The first step involves storing video frames as in block 400 of Fig. 4. 
The store of video frames will be stored in a database as set out in Table I, 



.(Attorney Signature) 


Mark A. Haynes, Reg. No. 30,846 
Signature Date: January 6, 1993 


PRELIMINARY AMENDMENT 


AMENDMENTS 
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entitled Database Frame Location below. As can be seen in the Table l.frames 
are stored in sequences which consist of an expression which proceeds from 
note Cm to note Cm#7, corresponding to a transition from event E1 to event 
E2. This consists of frames 1 -420. Each of the 1 0 expressions in the database 
are stored. 

in the next step, a key is assigned to each video frame corresponding 
to block 401 of Fig. 4. This is done by searching the database frame location 
database for the events E1 through E1 1 . According to this step, a table such 
as Table III, entitled Record Database, set out below is computed. Thus, the 
first entry in the Table I corresponds to the first entry in Table III. The first 
event E1 is in original frames 0-60. The transition from E1 to E2 is in original 
frames 60-360, and the second event stored in frames 360-420. This sequence 
is divided so that new frame numbers 0-220 are stored with a key identifying 
them as transition E1 to E2. 

In the next step, a content image as shown in Fig. 6 is generated based 
on the keys. This content image is generated by signing each key a position 
on the content image as shown on Table II below. Thus, event E1 will be 
assigned X,Y position (0,160) on the XY scale. Table II also shows the 
number of frames between each event on the display. 

The final step, the video frames are compiled for addressing in 
response to the key corresponding to block 403 of Fig. 4. This step of 
compiling video frames results in an assembled database as shown below in 
Table IV. A given frame, e.g. frame 0 is compiled so it can be accessed in 
response to position (0,160) on the frame. In this embodiment, a sequence 
of frames is displayed in response to a user selecting the position which range 
from frame 0-220 as shown in Table II. In a more simple case as described 
above, only a single frame may be displayed in response to positioning of the 
cursor. 

-2- 
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Once the content image of Fig. 6 is displayed and the assembled 
database of Table IV is computed, a user input devise may be used for 
accessing specific frames in response to a position on the content image as 
described with reference to block 404 of Fig. 4. 

These processing steps are thus executed by the processor 201 of Fig. 
2 and interconnected by the databases and tables computed by the 
processing steps. -- 

On page 25, line 14, before "Database Frame Location" please insert - Table I. - 

On page 26, please delete line 2 and insert therefore: 

-- The special coordinates of the content image are defined below in Table II. 

Table II. - 

On page 26, line 19, before "Record" insert ~ Table III. - 

On page 26, line 35, before "Assembled" insert - Table IV. - 

On page 26, after the last line, please insert the following: 

- In Table IV, the first column corresponds to a given node, e.g. node 
E1 in the database. The second column identifies the number of nodes that 
are associated with the node in column 1 . The third column identifies one of 
the adjacent nodes. The fourth column identifies the number of the frame 
associated with the first adjacent node identified in column 3. The fifth column 
identifies a second adjacent node, if any. The sixth column identifies the frame 
number associated with the second adjacent node. The seventh column 
identifies the X, Y coordinates on the content image of the node of column 1 . 
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The eighth column identifies the audio which is to be produced in association 
with the node in column 1 . 
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REMARKS 

In this Preliminary Amendment, the specification is amended to more clearly specify 
the use of the tables described on pages 25 and 26 of the specification in the context of Fig. 
4 of the present invention. 

The Examiner had issued a final action in the parent case on October 6, 1992. In 
such Official action, the Examiner objected to the drawings; rejected the specification under 
35 U.S.C. § 1 12, first paragraph; rejected claims 1-4, 6 and 7-10 under 35 U.S.C. § 1 12, first 
paragraph; rejected claims 1 , 2, 7, 8 and 9 under 35 U.S.C. § 102(b); rejected claims 3, 6, 10 
and 13 under 35 U.S.C. § 102(e); and rejected claims 4 and 11 under 35 U.S.C. § 103. 

Each of the Examiner's objections and rejections from the parent application is 
addressed below. 

Objection to the Drawings and Specification 

The Examiner objected to the drawings and specification as not showing claimed 
features of the present invention. 

As mentioned above, Applicant has amended the specification to more clearly specify 
the use of the data by the processing modules to apply the example system discussed on 
pages 25 and 26 of the specification. It is believed that this will more clearly specify how the 
modules, which are claimed in the present invention, are interconnected. This amendment 
also will enable the Examiner to recognize the components of the invention which are shown 
in the Figures and how they are interconnected. 

No new matter has been added, as the added text merely clarifies existing disclosure. 

Thus, reconsideration of the objection to the drawings and the specification under 
35 U.S.C. § 112, first paragraph, is requested. 
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Rejection of All Claims Under 35 U.S.C. § 112. first par agraph 

The Examiner rejected all claims under 35 U.S.C. § 112, first paragraph, 
as not enabled. It is submitted that the amendment to the specification set out above should 
address the Examiner's concern. Accordingly, reconsideration of the rejection of all claims 
under 35 U.S.C. § 112, first paragraph, is respectfully requested. 

Rejection of Claims 1. 2. 7 ft and 9 Under 35 U.S.C § 102(b) 

The Examiner rejected claims 1 , 2, 7, 8 and 9 under 35 U.S.C. § 102(b), as anticipated 
by Naimark, et.al. Understanding of the invention, the Examiner should reconsider the 
rejections. Naimark, et al. does not provide means for automatically assembling the 
database for the content image. Rather, in that system the content image was assembled 
manually and the frames associated with that content image by manual tabulation. It is that 
problem that the present invention solves. 

Accordingly, reconsideration of the rejection of claims 1, 2, 7, 8 and 9 under 35 
U.S.C. § 102(b) is respectfully requested. 

Rejection of Claims 3. 6. 10. and 13 Under 35 U.S.C. § 102(e) 

The Examiner rejected claims 3, 6, 10 and 13 Under 35 U.S.C. § 102(e) as anticipated 
by Morgan. It is believed that upon better understanding of the claims in view of the 
amendments set out above, the Examiner should withdraw this rejection. The Morgan 
reference cited by the Examiner has no connection with finding video after it has already 
been stored in a database. Rather it is only concerned with positioning a camera and 
controlling a camera position to shoot a particular image. Thus, reconsideration of the 
rejection based on Morgan is requested. 

Rejection of Claims 4 and 11 Under 35 U.S.C. § 103 

The Examiner rejected claims 4 and 11 Under 35 U.S.C. § 103 as obvious in light of 
Morgan and the Toshiba robotics paper. It is submitted that these claims clearly distinguish 
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over the combination of references for the same reason as there base claims distinguish over 
Morgan. Accordingly, reconsideration is requested. 

CONCLUSION 

It is submitted that the present application is now in form for allowance. No new 
matter has been added to the present application, as Applicant has merely clarified the 
description already present. 

The Commissioner is authorized to charge any underpayment or credit any 
overpayment to Deposit Account No. 06-1325 for any matter in connection with this 
response, including any fee for extension of time, which may be required. 


Respectfully submitted, 



By:. 



Mark A. Haynes 
Reg. No. 30,846 


FLIESLER, DUBB, MEYER & LOVEJOY 
Four Embarcadero Center, Suite 400 
San Francisco, California 94111-4156 
Telephone: (415) 362-3800 
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APPARATUS AND METHOD FOR ASSEMBLING 
CONTENT ADDRESSABLE VIDEO 

CROSS REFERENCE TO RE LATED APPLICATIONS 

The present application is related to U.S. Patent No. 
4,857,902, entitled POSITION- DEPENDENT INTERACTIVITY SYSTEM FOR 
IMAGE DISPLAY , invented by Naimark, et al . . issued August 15, 
1989; and to U.S. patent application Serial No. 07/356,543, 
entitled CONTENT ADDRESSABLE VIDEO SYSTEM FOR IMAGE DISPLAY, 
filed 89/05/24. 

FIELD OF THE INVENTION 
The present invention relates to systems for generating and 
storing video frames of information, such as produced using 
video/film cameras. In particular, the present invention 
provides a method and an apparatus for assembling frames of video 
data so that it may be accessed in response to the content of the 
respective frames . 

BACKGROUND OF THE INVENTION 
Interactive video systems, such as described in the above 
cross referenced U.S. Patent No. 4,857,902, in which a first 
monitor displays a virtual space which indicates the content of 
a library of video frames, and a second monitor displays video 
frames, in response to the position of a cursor on the virtual 
space, provide a powerful system for accessing video frames based 
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on the content of the video frames. The content is indicated by 
the virtual space. To access video based on content, a user 
moves a cursor to a position on the virtual space. In response 
to the position of the cursor on the virtual space , a video frame 
5 is accessed which has a content indicated by the position of the 
cursor . 

A problem has arisen in compiling video frames for the 
purposes of addressing them by content for use in such systems. 
In particular, the video frames are generated in such systems 
s=40 based on a plan organized by a human producer. The film is then 
% shot, such as by flying a helicopter on a geographic grid while 

:% filming a center point of reference. For instance, in the one 

^ system, a helicopter flew over the Golden Gate Bridge in San 

Francisco, while focusing the camera on the center of the bridge. 
Ml 5 Once the film has been generated, it must be manually 

compiled into sequences of video frames which correspond to the 
e ; grid flown over trie city. The grid is then drawn in a bit map 

graphic form for display as the virtual space. A correlation of 
the virtual space to the video frames is done manually in a time- 
20 consuming, tedious process. 

It is desirable to provide a method for compiling or 
generating content addressable video automatically. 
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SUMMARY OF THE INVENTION 

The present invention provides an apparatus and method for 
assembling content addressable video which is based on storing 
a plurality of frames of video data at addressable storage 
locations. Each frame of video data is stored with a tag which 
indicates the contents of the video image defined by the 
associated frame. A processing unit, assembles a content video 
image in response to the tags, including positions in the content 
image for corresponding frames of video data. Finally, a means, 
such as a look-up table, is provided for associating the 
positions in the content video image with addresses of storage 
locations storing the corresponding frames of video data. A user 
input device is provided by which the user selects a particular 
frame of video data by selecting a position in the content video 
image, such as by positioning a cursor on the selected position. 

The assembly of the content video image can be understood 
by considering the following example. If a plurality of frames 
of video data comprise film of a geographic area, each frame can 
be tagged with a position stamp, indicating the position of the 
camera when it was taken, and the focal point of the camera. 
This position stamp can then be translated automatically to a 
virtual position on a map. The map superimposed with the virtual 
positions of all the video frames in the plurality then 
constitutes the content video image. This content video image 
can be displayed using a bit map graphic system, in which a 
cursor can be positioned on any position in the content video 
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image. The position of the cursor is then translated into the 
position stamp of the frame of video data, and from the position 
stamp, into an address for the frame. 

According to another aspect, the present invention comprises 
an apparatus or method for generating content addressable video / 
which is the converse of assembling the content addressable 
video. In particular, according to this aspect, the content 
video image is first generated. Positions in the content video 
image are then translated by a control circuit into camera 
positioning signals. A controllable camera, such as a robot 
mounted camera , then generates the frames of video in response 
to the position control signals derived from the content video 
image. A processing unit then associates each frame of video 
data generated by the controllable camera, with positions in the 
content video image. 

The generation of content addressable video according to 
this aspect of the invention can be understood by considering the 
following example. Assume that one wanted to generate a content 
addressable video -library of an object, such as the transmission 
of an automobile. A content video image would then be 
graphically created in which an icon representing the 
transmission is placed in a virtual graphic space. A grid, three 
dimensional, or two dimensional, is then drafted using computer 
software graphics packages, to create a grid indicating the 
perspective of the transmission to be filmed. Thus, each 
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position in the grid would correspond to a frame of video data 
depicting a .different perspective of the transmission. 

A cursor is then scanned along the grid, either manually or 
using automatic computer programs, and cursor position is then 
5 translated into a camera control signal. The camera, mounted 
on a robot, then traverses real space corresponding to the 
virtual space of the content video image in response to the 
control signals, while filming the transmission. A processing 
unit then associates a position tag with each frame of video, and 
10 correlates the position tag with a position in the content video 
image. 

According to the invention, a user input device is provided, 
by which a user selects a particular frame of video by 
positioning a cursor on the content video image. This allows for 
15 highly interactive video systems in which frames of video data 
are addressed by content in an easily understandable and fun to 
use manner. 

Other aspects and advantages of the invention can be seen 
upon review of the drawings, the detailed description, and the 
20 claims which follow. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a schematic diagram of a system for generating, 
or assembling content addressable video according to the present 
invention. 

5 Fig. 2 is a schematic diagram of a system for interactively 

displaying content addressable video according to the present 
invention. 

Fig, 3 is a flow chart illustrating steps in a method for 
generating content addressable video according to the present 
10 invention. 

Fig. 4 is a flow chart illustrating the steps in a method 
for assembling content addressable video according to the present 
invention. 

Fig. 5 is a diagram of the assembly algorithm module for an 
15 uncontrolled camera environment. 

DETAILED DESCRIPTION 

A detailed description of preferred embodiments of the 
present invention is described with reference to the figures. 

20 In Fig. 1, a system for assembling or generating content 

addressable video is illustrated. The system includes a computer 
100 which provides processing capability used in assembling or 
generating the video. A computer is coupled to a camera position 
control robot 101 or other mechanism for controlling the position 

25 of a camera. The camera position is controlled to generate a 
plurality of frames of video information of an object 102 in a 
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space. Video frames are then stored in video storage 103* A 
video monitor 104 is provided for displaying the video frames as 
they are being filmed, or for accessing the video storage 103 to 
review the film. A content monitor 105 is provided which 
includes a graphic film content image 106. The film content 
image includes a depiction 107 of the object to be filmed, and 
a grid 108 having a plurality of positions . Positions on the 
grid 108 indicate the content of corresponding video frames in 
the video storage. Thus, as the camera position control robot 
101 moves the camera 109 along a plurality of camera positions 
indicated by the grid 110, the computer assigns a key to each 
frame of video. This key is correlated with a position in the 
graphic film content image 106 along the grid 108. 

A user input device 111, such as a mouse or track ball in 
the preferred system, can be used to position a cursor icon 112 
along the grid 108 in the graphic film content image 106. Thus, 
the position of the cursor icon 112 indicates the content of a 
frame of video being displayed on the video monitor 104. 
Therefore, for the object 102 being filmed from camera position 
as indicated at 109, and as represented by the cursor icon 112 
on the content image 106, an image would be displayed in the 
video monitor 104, -which shows the bottom, left side, and front 
of the object 102, as schematically illustrated in Fig. 1. 

The system can be expanded to include a plurality of cameras 
for filming a specific object, such as an automobile 
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transmission. Also, the object 102 can be moved instead of the 
camera to generate the frames of video. 

The content image 106 can be generated before filming the 
object 102, and thereby used to generate camera position control 
signals to control the generation of video. Similarly, the film 
can be shot first, and keys associated with each frame of video. 
The keys can then be analyzed by the computer 100 in order to 
generate a content image 106. This content image 106 can be 
generated at the same time that the camera is being used to film 
the object, or it can be generated on an as needed basis from a 
huge library of video frames. 

Thus, the system of Fig. 1 can be generalized by providing 
cameras attached to control arms that can move within a confined 
space, A virtual map or content image of frame content is mapped 
in a graphic image displayed on a content monitor. Control of 
the cameras, and their movements and point of view, are functions 
of positioning of camera icons or cursors in the graphic content 
image . 

Fig. 2 illustrates the preferred system for displaying the 
content addressable video according to the present invention. 
In this system, a content monitor 200 is provided which is 
connected to the computer 201. Two video disk players 202 and 
203 store a plurality of video frames. A video switcher 204 
which is controlled by computer 201 selects output video from the 
video disks in the players 202 and 203. The output video 
supplied to video monitor 205. A cursor control input device 
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206, like a mouse or track ball in the preferred system, is 
coupled to the computer 201. A user interactively positions the 
cursor 207 within the content video image 208 on the content 
monitor 200. A perspective of the object is displayed on the 
5 video monitor 205 which is indicated by the position of the 
cursor 207 in the content image 208, Thus, for the virtual space 
content image 208 illustrated in Fig. 2, a frame corresponding 
to the position of cursor 207, will provide a view of the object 
illustrated in the content image 208, which shows the bottom 
10 side, left side, and front side, as schematically illustrated in 
Fig. 2. 

Background concerning how to implement these interactive 
display systems can be found in the above cross-referenced U.S 
Patent No. 4,8577902. 

15 Fig. 3 illustrates the flow chart for the method using the 

apparatus of Fig. 1, or similar apparatus, to generate content 
addressable video. According to this method, a user uses a 
computer software graphics package or other tools to generate a 
content image based on desired video frames to be generated 

20 (block 300) . 

Next, the computer 100 is used to assign keys to positions 
in the content image. For the content image illustrated in Fig. 
1, these keys would correspond to the X, Y, and 2 positions on 
the grid 108 (block 301). 
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Next, the keys are used by the computer 100 to generate 
camera position control signals which control the robot. to film 
the object in response to the keys (block 302). Next, the keys 
are stored with or otherwise associated with each generated video 
image (block 303). The video frames are then compiled in a 
format whicl> facilitates accessing in response to the keys (block 
304). The step of compiling involves assigning the video frames 
to addresses in the video storage, and providing a means for 
translating the keys to the address of the corresponding video 
frame. These are standard computer techniques which can be based 
on look up tables and the like. Also, this compiling step 
involves identifying the frames of video on intersecting segments 
of the grid 108 and storing them on video disks, or other storage 
means, in storage positions which are accessible within a seek 
time at least as fast as the update rate of the video monitor, 
In the preferred system, this is at least 15 times per second. 

Finally, a user input device is provided for selecting a 
video frame in response to a position on the content image (block 
305 ). Again, this is provided by positioning a cursor on the 
content image using an input device. The position of the cursor 
then identifies a key which can be translated to an address of 
the corresponding video frame. Using the address, the video 
frame is accessed and displayed. 

Therefore, the content of the video frames is used to access 
25 them in a straightforward, highly interactive manner. 

Furthermore, this content image is used to automatically control 


15 


20 
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the generation of the video frames to be accessed in this content 
addressable video scheme. 

Fig. 4 is a flow chart illustrating the steps used for 
assembling cor;tent addressable video from a large library of 
5 video frames. 

This method involves first storing a library of video frames 
(block 400 Such video frames may, for instance, correspond 
to film of a highway system. 

Next, a key is assigned to video frames or segments of 
0 frames that are stored. These keys may correspond to geographic 
position stamps of the location filmed in the corresponding video 
frames (block 401). These keys can be calculated using 
interpolation techniques in which a sequence of video along a 
given highway, having a known start point and stop point can be 
5 assigned keys based on the estimated position of each frame. 

The next step involves generating a content image based on 
the assigned keys (block 402). According to this method, the 
assigned keys are used to compile a map using bit map graphics 
or the like, in which each key has a corresponding position on 
:0 the map. This content image can be generated for a subset of the 
entire library video frames, which consists of a few video frames 
or of the entire library. 

The next step involves compiling the video frames for 
addressing in response to the assigned key (block 403). The 
15 generated content image is used to compile the video frames by 
identifying positions on the content image of adjacent keys, and 
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storing the video frames at addresses indicated by the keys. For 
a large library of frames, a subset of the library which 
corresponds to at least a section of the content image is moved 
into a storage buffer that has reasonable access times, such as 
a writable optical disc or array of RAM. 

Finally, a user input device is provided for selecting a 
video frame in response to a position on the content image (block 
404). Having assembled a subset of the video frames by 
generating a content image representative of the content of the 
subset, a user can use a highly powerful interactive scheme for 
positioning a cursor on the content image to select the 
particular video frame having the desired content. 

The basic tools and methodology for the assembly of content 
addressable video systems are very similar for the various data 
15 acquisition methods. The process involves the marriage between 
a graphic space, acquired visual data and the relational content 
image of this visual data in the graphic space. in the system 
there exist at least an element of organization, i.e. , a physical 
location of visual data and its respective position in a graphic 
20 space (content image). This organization is critical to the 
representation of information and serves as the base methodology 
for a visual language. Content Addressable Video systems may be 
built from closed systems, controlled camera environments, and 
uncontrolled or free camera environments. They may be built from 
25 a combination of aj.1 three. 
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A. CLOSED SYSTEM EWTRONMPMT 

A closed system implies a graphic or artificial environment. 
An example of such a system is a CAD workstation which has tools 
for designing objects as well as for the three dimensional 
animation of these objects frame by frame into a movie. Software 
provides the producer the capability to build the content image 
into the graphic space. The tools for drawing a content image 
already exifct in CAD systems. 

Once the content image is constructed, a light is projected 
from a cursor which represents the focal plane of the camera. 
A joystick controls the direction of the camera (cursor or camera 
icon) while the cursor traverses the content image in preview 
mode. This provides for accurate pan, tilt keys as well as 
accurate position data. Once the operator decides this content 
image and camera point of view are correct, he implements an 
"animate" command and each frame is rendered, stored and finally 
compiled for interactive control. Hence, a system as described 
in Fig. 2 with the CAD and animation tools with the content 
addressable image software technology. 

The key stamps consist of position, pan, and tilt with their 
respective video frame derived directly from their location in 
the graphic virtual space. 

This module provides segments or geometric shapes to be 
recognized by the camera paths. These shapes may be drawn with 
the normal draw routines. The program will ask for the frame 
density along the paths and the camera POV. The line segments 
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will parse frame locations along the line segments and send keys 
out to a frame database and then calculate the frame by frame 
animation. The frames are stored to optical discs or other 
memory, such as large 3-D arrays of DRAMs or VRAMs , accessed in 
a massively parallel means, and the content image is assembled. 
The matrix arrangements are compiled and a cursor appears on the 
content image in the graphic space once the compilation and 
assembly is complete providing control to the user. 

B- CONTROLLED CAMERA E NVIRONMENT 

In this method of data acquisition, the visual data is 
acquired in controlled filming environments; specifically those 
where cameras are controlled in stationary or motion controlled 
environments. The assembly of these content addressable video 
systems will require predefined matrices of film maps generated 
on a graphic computer platform which translates actual commands 
to a camera motion control platform for videomap production and 
automated' assembly. In this environment, the space which the 
cameras can cover are scaled to the virtual graphic space such 
that an incremental movement of a cursor on a content image 
solicits a relative movement of the camera (or video once shot 
and compiled) in its respective space. 
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In our system; we will place the subject, say an automobile 
transmission, on the platform. 

The algorithm works as follows: 

1. Scale Camera Locations in real space to camera locations 
in the graphic content image space. Also zero out camera pan, 
tilt and direction POV. Position subject in content image space. 

2. Locate graphic model center at x, y, z in graphic 
virtual space on motion platform. 

3. Build the content image in the graphic space using the 
graphic software tool commands. Or build sequence in real space 
using actual cameras. The actual camera moves are recorded, 
stamped, and converted to the graphic space. 

4. Stamp keys on frames (or other recording methods 
discussed below) by recording camera position, POV, Pan, and Tilt 
in real time while recording frames. 

5. Record filmed frames to optical disc player or to other 
archive technology . 

6. Convert camera position stamp locations to content image 
scale and location. Store key data locations of camera in the 
content image space frame by frame. Location data consist of x, 
Y, z location, Pan, Tilt, and focal plane. 

7. Store locations of frames in content image space (x, y, 
z) relationship of each frame to its corresponding content image 
in a graphic space using keys and the system 7 s processing unit. 
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8. Sort and determine the visual matrix as well as 
relationship of each frame to its corresponding content image in 
a graphic space using keys and the system's processing unit. 

9. Compile database in its matrix structure. 

10. Generate content image space in virtual space. 

11. Choose starting point in matrix. 

12. Compile and organize frame data and reference content 
image space relative to virtual space. 


10 Camera Syst.Pj 

The way it works is as follows: 

Cameras are attached on control arms and can move anywhere 
in one virtual space. A content image is drawn in response to 
the camera movement or by hand in the graphic virtual space 
15 (computer screen) such that an incremental move or matrix 
location in the camera space is equal to the incremental vector 
in the graphic space. In other words, from picture to picture 
the video matrix corresponds to its content image in its graphic 
space. 

20 

VNCQNTEQEfcEP ( FKKK ) CAMERA E NVIRONMENT 

The third data acquisition environment deals with data 
acquired in uncontrolled filming environments. This acquisition 
requires accurate position information and Pan (P), Tilt (T) as 
25 frame stamping as keys for content image assembly. These video 
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matrices are assembled into accurate survey mapping systems as 
well as other GIS systems for display and overlay capability. 

Fig. 5 is a diagram as to the modules of the code in the 
assembly algorithm. This system integrates an uncontrolled 
platform (free) or other film systems for videomap production. 
There is no limit placed upon the size of matrices. This may 
require multiple stores of optical discs. In this case, the 
database structure requires an additional data location stamps 
for archive purposes. 

Aerial or Free Camera Systems 

In the case of free filming camera systems, (i.e., cameras 
in helicopters, balloons, Remote Piloted Vehicles, etc.) a final 
post processing step must be implemented to compile accurate 
relationships between the content image space and its relative 
virtual space. The method of scripting of content addressable 
video systems provide for a combination of free-form data 
acquisition and tight grid or matrix data acquisition. 

The following additional aspects to this workstation are 
important for flying or developing videomaps for uncontrolled 
spaces . 

1. Camera speed should be directly controlled to provide 
an accurate matrix when using a free virtual filming system such 
as a helicopter, balloon, ground gyrostabilized camera mount such 
as a steadycam etc. This provides compensation for environmental 
effects of wind and other factors which affect camera position. 
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2. Actual location of the camera is determined by updates 
of camera location, pan, tilt axis on a frame by frame basis. 
This information is processed back to the navigational controls 
of the camera platform (including helicopter) in order to provide 
corrections and adjustments in direction location and POV. 

3. The system is designed with a predetermined content 
image structure prior to recording video or film. This content 
image structure produces a navigational data set for the camera. 
This navigational information incorporates supporting flying 
surfaces such as those used to guide the flying machine. 

Design of an Aarial V-trtartnmp 

The design requires a map system which contains actual 
spatial elements of the area which is to be mapped. Since the 
current filming system is by helicopter and much of the data 
acquisition is implemented by sight flying with some GPS 
navigation aid. A remote powered vehicle drone is utilized for 
"close-in" mapping. 


Position Stamni nrr 

The current method of frame location in video is a standard 
called SEMPTE Time Code. Its structure is as follows: 
Hour, Minutes, Seconds, Frames 
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In the preferred embodiment, this code is expanded into the 
following structure : 

Hour, Minutes, Seconds, Frames, Earth Reference Fixed GPS Receive 
Position, x, y, z, position GPS (Time T) , Camera Receiver 
Position x, y, z GPS, Velocity (Time T) , Acceleration Time (T) , 
Corrected Compass Camera Mount Facing (Time T) , Pan degree (Time 
T), Tilt (Time T) , Rotation (Time T) , Frame Number and the 
content image scale to the graphic space (map) . 

The sample rate of the data will be controlled by the camera 
frame rate or speeds which determine frame accurate stamping. 
As frames are shot, they will be written to the nearest sample. 
Current GPS data is available on one second intervals but may 
change given satellite positioning and reception quality. Post 
processing is currently necessary to average distance between GPS 
position samples, distance covered and number of frames per 
second. Thus, location stamping requires post processing in this 
situation. The satellite locations are scrambled and also have 
to be post processed . 

The on-board filming system is designed to send data to a 
counter and then to the data accumulator per frame of video or 
film. If film is used, each reel will have its own position 
database. The position stamps are included with each individual 
frame. Currently, production technology provides a standard way 
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of adding SEMPTE time code to video. Position stamping according 
to the present invention may follow the same standard 
methodology . 

Methods Qf Storing Stamps and PrP.sn r ibPd n a t- ft !? ftys Aro . 

1. Actual time stamping and a comparison algorithm of 
camera location at time (T) and frame exposure time (T) . 

2. Writing actual time and position data on sound track 
of video. 

3. Use of computer memory for storing frame exposure time 
and position simultaneously. 

Trimble Navigation provides the Trimble Advanced Navigation 
Sensor System for GPS. Our system will use this technology or 
some other GPS system which is designed to triangulate satellite 
signals for position. A second ground station receiver will also 
be used to improve accuracy and the data collected and merged 
with the data received (sample rate) in the aircraft and 
processed to refine error rates and improve position accuracy to 
an area of 5 meters cube or better. 

Another system is currently being designed which utilizes 
a mirror system on a drone PRV and a laser. The laser is 
directed at the PRV and a receiver whose position is accurately 
surveyed, processes the laser band reflection and samples actual 
RPV location for real time positioning. This system and position 
stamp per frame will be similar, however, it will provide very 
accurate positioning. 
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A software routine will build a system of frame location 
the three dimensional virtual space based upon the following 
organization. 
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Content Addressable Vid^n Asspmhly 

The following treatment of the software tools which may be 
used to develop a prescribed sequence of routes and camera 
position which when implemented will provide an autopilot 
control, camera position (pan and tilt) and frame controlled for 
the data acquisition hence the tools for an assembly language. 

The elements of this system are: 

1. Accurate maps of prescribed videomap area. 

The U.S. Geological Survey (USGS) has accurate mapping maps 
in vector form" of many areas which have been processed by 
satellite. The design of a videomap begins with this information 
in a workstation. It is not essential to have this information 
for the design because the data acquisition position stamping 
will provide the information necessary to merge this information 
into these map databases at a later date. However, all 
productions begin with a script or plan based upon a desired 
application and the sequences of visual material which best fit 
the application. Obviously, there are many accurate methods of 
obtaining pictures of the ground; they include SPOT Satellite, 
Space Shuttle picture systems and high reconnaissance aircraft. 
A Moviemap or Videomap of these pictures and their translation 
into continuous imagery via a continuum of connected images in 
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25 FPS or 30 FPS or some visual translation from frame to frame 
in a raster frame merger is another option and used in a closed 
system . 

Videomap Production 

As mentioned above, the beginning design requires some 
element of production scripting and design. The GPS system will 
also be designed as a pilot aid for navigation. The designer 
begins by using the Content Addressable Video workstation to 
design his production, that is, the content image to scale in the 
prescribed graphic space. A location cursor in the aircraft in 
conjunction with the GPS hardware and a two dimensional map with 
the actual routes to scale will provide the pilot an on-board 
navigation system. This is integrated with the autopilot. 
Though a good quality structure improves the overhead processing 
requirement's and delta graphic overlay displays it is not 
essential to get it exact. However, it is important to know 
precisely the position of what was recorded. Pilots are very 
capable of flying VFR and instruments accurately with 
20 navigational assistance. 


10 


15 


25 


D - PATA , ACQUISITION AND ASSEMBLY O F A CCWPPMT ADDRESSABLE VIDRO 

The assembly techniques can be used in a variety of 
contexts. For example, an automatic assembly routine can be used 
for teaching music. In the music example: One might film someone 
playing a song on a piano (hands only). The script of the song 

- 22 - 
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may be broken down to sound and visual entities of frames of each 
depression of the keys. Each chord and the associated frames 
would be an event. The assembly of two chords together would be 
an expression of events, the multiple expression of events would 
5 result in the script. 

So the fundamental logic or algorithms behind the Content 
Addresssable Video System are driven and organized in this 
example by this fundamental structure. 

This example assumes a database of extensive events and a 
0 logical form of transition. It is essential to include a 
distinction between the actual tools of organization and assembly 
of data and the creative elements which provide a transition 
continuity to the data. 

A database must have a multiplicity of creative transitional 
.5 elements. Current post production equipment has so many "hacks' 1 
to handle a* lack of these transitional elements. A leading film 
maker established the use of "special transition effects" as 
transitional elements. Hitchcock was a master of the study of 
transitional elements . 
10 An interactive movie must contain events. These events must 

be expressed. The interactive dimension is established through 
creative transitional elements as illustrated in Fig. 5. 

Let's design an interactive piano and then assemble a song 
using this method. 
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I. Develop the database for the Script 

A. Define the number of events 500-1 through 500-6.. These 
events are all chords in the music spectrum. 

B. Create the music for each event and store the sound 
5 data chord (E - time). 

C. Develop transitions between Events, Film expressions 
(e.g., 501-1 through 501-3) of all combinations of two events. 
That is, all of the independent physical hand movements from 
chord event to chord event. 

10 II. Write the script 

A. "This Masquerade" cl973 Teddy Jack 

c ™ Cm#7 Cm7 F9 

we really hap££ with this lonely game we plav? . . . . 

Crn Ab7 G7 Cm Cm#7 

Inking for words £a say /JUJ _ s^rching but not finding; 

Cm7 F9 
understanding any way . " 

B. List Events - (Underlines indicate location of events.) 

There are 11 events proposed in this song. They are: 

El - Cm 
E2 - Cm #7 
25 E3 - Cm7 

E4 - F9 
E5 - Cm 
E6 - A b7 
E7 - G7 

30 E8 - Cm 

E9 - Cm #7 
E10 - Cm7 
Ell - F9 


15 


20 


35 


III. Now implement the automated assembly function. 

End result for preview is illustrated in Fig, 6. 
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First frame of hands in the Cm playing position El. System 
prepared to cue first Expression E1-E2 (Cm-Cm#7) transition is 
the hand movement from Cm to Cm#7. Interactive control with 
cursor ( * * * ) . 


Sound Data and Relational Data Location 

Sound of Cm chord based upon movement of location cursor 

( * * * ) 


Content Addressable Video System Process (Automated Procedure) 
Step 1 

Search database for El-Ell Expressions 


Database Frame Location (arbitrary for this example) 


15 



Event 

Transition 

Event 


gxpr^ssipn 

Frames. 

Frames 

Frames 


1 . 

Cm to Cm#7 

1-60 

60-360 

360-420 


2 . 

Cm#7 to Cm7 

1500-1560 

1560-1860 

1860-1920 


3 . 

Cm7 to F9 

2100-2160 

2160-2460 

2460-2520 


4 . 

F9 to Cm 

3000-3060 

3060-3360 

3360-3420 


5 . 

Cm to A b7 

3500-3560 

3560-3860 

3860-3920 


6 . 

A b7 to G7 

4000-4060 

4060-4360 

4360-4420 

25 

7 . 

G7 to Cm 

5000-5060 

5060-5360 

5360-5420 

8 . 

Cm to Cm#7 

6000-6060 

6060-6360 

6360-6420 


9 . 

Cm#7 to Cm7 

7000-7060 

7060-7360 

7360-7420 


10 . 

Cm7 to F9 

8000-8060 

8060-8360 

8630-8420 


In this example , the : 

Event Time (t) is 60 frames or 2 seconds 
Transition (t) is 300 frames or 10 seconds 
Expression (t) is 360 or 12 seconds. 


Determine location of all underlines in x, y screen 
coordinates of content image. In this case, the words designate 
the map distance between events. In this example, the 
differential of the x, y coordinates between events provides the 
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transition scale for the expressions. Total line length x = 160. 



ea C b 

expression will 

contain 





y 

Dx 

Dv ( 1 ine 

mac ) 

I # Frames 

E 1 

o 

160 


0 



E2 

60 

160 

60 

0 

50/160( 300) 

94 

E3 

30 

160 

20 

0 

30/160(300) 

56 

E4 

140 

160 

80 

0 

80/160(300) 

150 

E5 

0 

130 

30 

0 

30/160(300) 

56 

E6 

60 

130 

60 

0 

60/160(300) 

112 

E7 

70 

130 

10 

0 

10/160(300) 

150 

E8 

80 

130 

10 

0 

10/160(300) 

19 

E9 

160 

130 

80 

0 

80/160(300) 

150 

E10 

30 

100 

30 

0 

30/160(300) 

56 

Ell 

80 

100 

50 

0 

50/160(300) 

94 


Record Database 

Skip Recorded 
Event: Transition- Frames b'vent N<?V Frajng # 


E1-E2 

0- 

60 

60- 

360 

3 

360- 

420 

0- 

220 

E2-E3 

1500- 

1560 

1560- 

1860 

5 

1860- 

1920 

221- 

400 

E3-E4 

2100- 

2160 

2160- 

2460 

1 

2460- 

2520 

401- 

670 

E4-E5 

3000- 

3060 

3060- 

3360 

5 

3360- 

•3420 

671- 

850 

E5-E6 

3500- 

3560 

3560- 

3860 

3 

3860- 

3920 

851- 

1070 

E6-E7 

4000- 

4060 

4060- 

4360 

1 

4360- 

4420 

1071- 

1340 

E7-E8 

5000- 

5060 

5060- 

5360 

15 

5360- 

5420 

1341- 

1480 

E8-E9 

6000- 

6060 

6060- 

6360 

1 

6360- 

6420 

1481- 

1750 

E9-E10 

7000- 

7060 

7060- 

7360 

5 

7360- 

7420 

1751- 

1930 

ElO-Ell 

8000- 

8060 

8060- 

8360 

3 

8360- 

8420 

1931- 

2150 


Assembled Database 


Node 

Num Nodes' Node 

Num Frame 

Num Node 

Frame 

X.Y Audio 

1 

1 

2 

0 



0,160 Cm 

2 

2 

1 

220 

3 

220 

60,160 Cm#7 

3 

2 

2 

400 

4 

400 

80,160 Cm7 

4 

2 

3 

670 

5 

670 

140,160 F9 

5 

2 

4 

850 

6 

850 

0,130 Cm 

6 

2 

5 

1070 

7 

1070 

60,130 Ab7 

7 

2 

6 

1340 

8 

13 4-0 

70,130 G7 

8 

2 

7 

1480 

9 

1480 

80,130 Cm 

9 

2 

8 

1750 

10 

1750 

160,130 C#7 

10 

2 

9 

1930 

11 

1930 

30,100 Cm7 

11 

1 

10 

2150 



80,130 Fb 
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E. MATRIX ASS EMBLY ALGORITHMS 

1. Beginning, End, or flaw in frame data algorithm 
Key stamps of the video data are read and stored into a key 
database. This is particularly useful and essential for the 
assembly of the video frame matrix, A number of algorithms may 
be processed during the read and store function from tape to the 
key database, including the following two important: components: 

1 . Relationa l Frame Matrix Algorithm 

This process* studies each frame and their keys to determine 
their matrix locations. This process establishes common frames, 
nodes, and the spatial relationships for the content image and 
its relational virtual space. 

2. Image Re cognition Kev 

During the process of recording from tape to key database 
an algorithm is used to study the differential in pixel change 
from frame to frame in real time. A key is stored when the 
differential is large enough from frame to frame to locate a flaw 
or edit or cut between sequences. This will aid in reaffirming 
the relational frame matrix algorithm. 

SUMMARY 

This technology automates the organization and assembly of 
video frame data such that the acquisition, assembly, and 
ultimate integration of images may be represented in a relational 
graphic space via a content image. These systems' real time 
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capability are a function of speed memory and memory burst rate 
and the platform will effect the performance. The display format 
of these large stores of archived image data, content images 
files and relational graphic spaces form a powerful interactive 
visual display language . 

The foregoing description of preferred embodiments of the 
present invention has been provided for the purposes of 
illustration and description. It is not intended to be 
.exhaustive or to limit the invention to the precise forms 
disclosed. Obviously, many modifications and variations will be 
apparent to practitioners skilled in this art. The embodiments 
were chosen and described in order to best explain the principles 
of the invention and its practical application, thereby enabling 
others skilled in the art to understand the invention for various 
embodiments and with various modifications as are suited to the 
particular use contemplated. It is intended that the scope of 
the invention be defined by the following claims and their 
equivalents . 

What is claimed is: 
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CLAIMS 

1. An apparatus for assembling content addressable video , 
comprising : 

storage means for storing a plurality of frames of video 
data in storage locations having addresses, each frame defining 
a video image having a content for display; 

means ; coupled with the storage means, foe associating tags 
with frames of video data in the plurality, the tags indicating 
the contents of the video images defined by the associated 
frames; 

processing means, coupled to the means for associating, for 
assembling a content video image in response to the tags, the 
content video image including positions for corresponding frames 
of video data in the plurality; and 

means, coupled to the processing means, for associating- 
positions in the content video image with addresses of storage 
locations storing corresponding frames of video data. 

2. The apparatus of claim 1, further including: 
means for selecting a position in the content video image; 
means, coupled with the means for selecting and the means 
for associating positions, for accessing the frames of video data 
in the storage means in response to selected positions. 
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3. An apparatus for generating content addressable video, 

comprising : 

means for generating a content video image representative 
of an organization of content addressable video, the content 
5 video image having positions within the content video image 
corresponding to desired content of video images to be displayed; 

control means, coupled to the means for generating, for 
generating control signals indicating a content for a video image 
in response to positions within the content video image; 
10 controllable means, responsive to the control signals, for 

generating frames of video data, each frame defining a video 
image having the content indicated by the control signals; and 
processing means, coupled to the controllable means and the 
control means, for associating frames of video data generated by 
15 the controllable means with positions in the content video image. 

4. The apparatus of claim 3, wherein the controllable 
means comprises a robot mounted video camera. 

5. The apparatus of claim 3, wherein the processing means 
comprises : 

storage means , coupled to the controllable means, for 
storing frames of video data generated by the controllable means 
5 in storage locations having addresses; and 
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means coupled to the controllable means and the control 
means, for associating the address of each frame of video data 
with a position in the content video image. 

6. The apparatus of claim 5, further including: 

means for selecting a position in the content video image; 

means, coupled with the means for selecting and the means 
for associating, for accessing the frames of video data in the 
storage means in response to selected positions. 

7, A method for assembling content addressable video, 

comprising : 

storing, in an addressable memory , a plurality of frames of 
video data in storage locations having addresses, each frame 
defining a video image having a content for display; 

associating tags with frames of video data in the plurality, 
the tags indicating the contents of the video images defined by 
the associated frames; 

assembling a content video image in response to the tags, 
the content video image including positions indicating the 
content of corresponding frames of video data in the plurality; 
and 

associating with data processing means the positions in the 
content video image with addresses of storage locations storing 
corresponding frames of video data. 
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8, The method of claim 7, further including: 

selecting, with a user input device, a position in the 

content video image ; 

accessing, with data processing means, the frames of video 

data in the storage means in response to a selected position. 

9. The method of claim 7, further including: 

storing in a cache memory a subset of the plurality of 
frames, the subset including frames having content indicated by 
at least a portion of the content video image. 

10. A method for generating content addressable video, 
compris ing : 

displaying a content video image representative of an 
organization of content addressable video, the content video 
image having positions within the content video image 
corresponding to desired content of video images to be displayed; 

selecting with data processing means positions within the 
content video image; 

generating control signals indicating a content for a video 
image in response to the selected positions within the content 
video image; 

generating frames of video data in response to the control 
signals, each frame defining a video image having the content 
indicated by the control signals; and 
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associating with data processing means the generated frames 
of video data with positions in the content video image. 

11. The method of claim 10, wherein the step of generating 
frames comprises: 

controlling a robot mounted video camera in response to the 
control signals . 

12. The method of claim 10, wherein the step of associating 
comprises : 

storing generated frames of video data in storage locations 
having addresses; and 

associating the address of each frame of video data with a 
position in the content video image. 

13. The method of claim 12/, further including: 
selecting with a user input device a position in the content 

video image ; 

accessing the frames of video data in the storage means in 
response to selected positions. 
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APPARATUS AND METHOD FOR ASSEMBLING 
CONTENT ADDRESSABLE VIDEO 
ABSTRACT 

A system and method for assembling or generating content 
addressable video based on storing a plurality of frames of video 
data at addressable storage locations. Each frame of video data 
is stored with a tag which indicates the contents of the video 
image defined by the associated frame. For assembly, a 
processing unit assembles a content video image in response to 
the tags; the content video image, including positions for 
corresponding frames of video data* Finally, a means, such as 
a look up table, is provided for associating the positions in the 
content video image with addresses of storage location storing 
the corresponding frames of video data- A user input device is 
provided by which the user selects a particular frame of video 
data, by selecting a position in the content video image, such 
as by positioning a cursor on the selected position. 

For generating content addressable video, the content video 
image is first generated- Positions in the content video image 
are then translated by a control circuit into camera positioning 
signals. A controllable camera, such as a robot mounted camera, 
then generates the frames of video in response to the position 
control signals derived from the content video image. A 
processing unit then associates each frame of video data 
generated by the controllable camera, with positions in the 
content video image. 
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FIGURE 6 

Content Image With Graphic Space 

Cm (****) Cm# 7 Cm7 F9 

Are we really happy with this lonely game we play? . . • 

Cm A--b7--G7 Cm Cm#7 

Looking for words to say ... searching but not finding 

Cm7 F9 

understanding anyway, . 
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Title 37. Code of Federal Regulations. $1.56 


SECTION 1.56. DUTY TO DISCLOSE INFORMATION 
MATERIAL TO PATENTABILITY 


(a) A patent by its very nature is affected with a 
public interest. The public interest is best served, 
and the most effective patent examination occurs 
when, at the time an application is being examined, 
the Office is aware of and evaluates the teachings of 
all information material to patentability. Each 
individual associated with the filing and prosecution 
of a patent application has a duty of candor and 
good faith in dealing with the Office, which includes a 
duty to disclose to the Office ail information known to 
that individual to be material to patentability as 
defined in this section. The duty to disclose 
information exists with respect to each pending claim 
until the claim is cancelled or withdrawn from 
consideration, or the application becomes 
abandoned. Information material to the patentability 
of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the 
information is not material to the patentability of any 
claim remaining under consideration in the 
application. There is no duty to submit information 
which is not material to the patentability of any 
existing claim. The dutyio disclose all information 
known to be material to patentability is deemed to be 
satisfied if all information known to be material to 
patentability of any claim issued in a patent was cited 
by the Office or submitted to the Office in the manner 
prescribed by §§1 .97(b)-(d) and 1 .98.* However, no 
patent will be granted on an application in connection 
with which fraud on the Office was practiced or 
attempted or the duty of disclosure was violated 
through bad faith or intentional misconduct. The 
Office encourages applicants to carefully examine: 

(1) prior art cited in search reports of a foreign 
patent office in a counterpart application, and 

(2) the closest information over which 
individuals associated with the filing or 
prosecution of a patent application believe any 
pending claim patentabiy defines, to make sure 
that any material information contained therein is 
disclosed to the Office. 


(1) It establishes, by itself or in combination with 
other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position 
the applicant takes in: 

(i) Opposing an argument of unpatentability 
relied on by the Office; or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established 
when the information compels a conclusion that a 
claim is unpatentable under the preponderance of 
evidence, burden-of-proof standard, giving each term 
in the claim its broadest reasonable construction 
consistent with the specification, and before any 
consideration is given to evidence which may be 
submitted in an attempt to establish a contrary 
conclusion of patentability. 

(c) Individuals associated with the filing or 
prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or 
prosecutes the application; and 

(3) Every other person who is substantively 
involved in the preparation or prosecution of the 
application and who is associated with the 
inventor, with the assignee or with anyone to 
whom there is an obligation to assign the 
application. 

(d) Individuals other than the attorney, agent or 
inventor may comply with this section by disclosing 
information to the attorney, agent, or inventor. 


(b) Under this section, information is material to 
patentability when it is not cumulative to information 
already of record or being made of record in the 
application, and 


* §§1.97(b)-(d) and 1.98 relate to the timing and 
manner in which information is to be submitted to the 
Office, 
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OR REJECTION OF APPLICATION . 

(a) A duty of candor and good faith toward the Patent and 
Trademark Office rests on the inventor, on each attorney or agent 
who prepares or prosecutes the application and on every other 
individual who is substantively involved in the preparation or 
prosecution of the application and who is associated with the 
inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. All such individuals have 
a duty to disclose to the Office information they are aware of 
which is material to the examination of the application. Such 
information is material when there is a substantial likelihood that 
a reasonable examiner would consider it important in deciding 
whether to allow the application to issue as a patent. The duty is 
commensurate with the degree of involvement in the preparation or 
prosecution of the application. 
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